Demographic Inference and Representative Population Estimates from Multilingual Social Media Data

TL;DR

This paper introduces a multilingual deep learning approach for demographic inference from social media data, combined with bias correction techniques, to produce more representative population estimates across diverse languages and regions.

Contribution

It develops a novel multimodal neural network for demographic classification in 32 languages and proposes interpretable multilevel regression for bias correction, advancing social sensing accuracy.

Findings

Outperforms state-of-the-art demographic inference methods.

Reduces algorithmic bias in demographic classification.

Enables more accurate population estimates in multilingual regions.

Abstract

Social media provide access to behavioural data at an unprecedented scale and granularity. However, using these data to understand phenomena in a broader population is difficult due to their non-representativeness and the bias of statistical inference tools towards dominant languages and groups. While demographic attribute inference could be used to mitigate such bias, current techniques are almost entirely monolingual and fail to work in a global environment. We address these challenges by combining multilingual demographic inference with post-stratification to create a more representative population sample. To learn demographic attributes, we create a new multimodal deep neural architecture for joint classification of age, gender, and organization-status of social media users that operates in 32 languages. This method substantially outperforms current state of the art while also reducing algorithmic bias. To correct for sampling biases, we propose fully interpretable multilevel regression methods that estimate inclusion probabilities from inferred joint population counts and ground-truth population counts. In a large experiment over multilingual heterogeneous European regions, we show that our demographic inference and bias correction together allow for more accurate estimates of populations and make a significant step towards representative social sensing in downstream applications with multilingual social media.

Tables

Table 1. Table 1. Sizes of datasets used for training and testing. Incremental numbers are shown for the outputs of data-augmentation from co-training and translation.

Dataset	Gender	Age	Is-Org.
Initial	3.98M	1.20M	59.92K
After Image Co-Training	+2.26M	+0.62M	+6.31M
After Translation Augmentation	+5.74M	+1.54M	+4.74M
After Text Co-Training	+8.28M	+0.78M	+11.87M
Final	14.53M	2.61M	23.86M
Held-out Evaluation	0.38M	0.36M	6.91K

Table 2. Table 2. Performance comparison on gender using text methods. † Demographer only has a coverage of 0.974.

Method	Macro-F1
M3: Full Text	0.907
M3: Username Only	0.828
GenderPerformr (Wang and Jurgens, 2018)	0.835
Demographer (Knowles et al., 2016)	0.781	†
Jaech and Ostendorf (2015)	0.763

Table 3. Table 3. Performance at recognizing organizational accounts, measured as accuracy per class, following (McCorriston et al . , 2015 )

Method	Person	Organization
M3	0.986	0.807
Demographer (Wood-Doughty et al., 2018)	0.973	0.644
Humanizr (McCorriston et al., 2015)	0.982	0.586

Table 4. Table 4. Performance (Macro-F1) for (1) full M3 model, (2-3) model with modality dropouts, (4) model trained without co-training or translation, (5-6) Random and and Majority-class baselines.

	Heuristically-labeled			Multilingual
M3 Model	Gender	Age	Org-Status	Gender	Age	Org-Status
Full	0.918	0.522	0.898	0.915	0.425	0.898
w/ Text Dropout	0.743	0.349	0.837	0.837	0.359	0.892
w/ Image Dropout	0.905	0.493	0.823	0.862	0.372	0.822
w/o Co-T. & Trans.	0.918	0.449	0.917	0.875	0.325	0.660
Random	0.494	0.181	0.500	0.494	0.211	0.466
Majority	0.377	0.214	0.500	0.334	0.187	0.210

Table 5. Table 5. Performance on the Gender Shades dataset (cf. Table 4 in (Buolamwini and Gebru, 2018 ) ). Gender classification performance as measured by the positive predictive value (PPV), true positive rate (TPR), and false positive rate (FPR).

Classifier	Metric	All	F	M	Darker	Lighter	DF	DM	LF	LM
MSFT	PPV	93.7	89.3	97.4	87.1	99.3	79.2	94.0	98.3	100.0
	TPR	93.7	96.5	91.7	87.1	99.3	92.1	83.7	100.0	98.7
	FPR	6.3	8.3	3.5	12.9	0.7	16.3	7.9	1.3	0.0
Face++	PPV	90.0	78.7	99.3	83.5	95.3	65.5	99.3	94.0	99.2
	TPR	90.0	98.9	85.1	83.5	95.3	98.8	76.6	98.9	92.9
	FPR	10.0	14.9	1.1	16.5	4.7	23.4	1.2	7.1	1.1
M3	PPV	96.7	99.4	94.5	94.0	99.3	99.6	89.3	99.3	99.2
	TPR	96.5	92.8	99.6	93.4	99.3	86.0	99.7	99.0	99.5
	FPR	4.2	0.4	7.2	7.7	0.8	0.3	14.0	0.5	1.0

Equations

$$N^i$$

$${\pi }^i(a,g)=M^i(a,g{)}^{\nu }{f}_1(a)f_2(g),$$

$$\log N^i(a,g)$$

Full text

Demographics; Post-stratification; Social Media; Latent Attribute Inference; Inclusion Probabilities; Multilingual; Deep Learning

††copyright: iw3c2w3††journalyear: 2019††conference: Proceedings of the 2019 World Wide Web Conference; May 13–17, 2019; San Francisco, CA, USA††booktitle: Proceedings of the 2019 World Wide Web Conference (WWW ’19), May 13–17, 2019, San Francisco, CA, USA††doi: 10.1145/3308558.3313684††isbn: 978-1-4503-6674-8/19/05††ccs: Human-centered computing Social media††ccs: Computing methodologies Natural language processing††ccs: Computing methodologies Object identification††ccs: Computing methodologies Neural networks

1. Introduction

Data representing the attitudes and behaviors of a (national) base population is of great importance to policy making, social science research and commercial prediction tasks, but representative surveys are expensive and infrequent. Social media data has been proposed as a real-time and inexpensive way to measure social phenomena—this area of research is known as “social sensing.” Social media data has been used, with various levels of success, in areas such as infectious disease (Ginsberg et al., 2008; Lazer et al., 2014; Generous et al., 2014), migration and tourism (Lamanna et al., 2018; Zagheni et al., 2017; Barchiesi et al., 2015), and box office takings for films (Mestyán et al., 2013; de Silva and Compton, 2014).

However, despite social media data being easily accessed, large in scale, and detailed, it is generally not representative of indicator variables measured in any broader (offline) population due to various biases (Ruths and Pfeffer, 2014; Jungherr, 2017). Especially the self-selection of users joining and participating on any given social media platform is a large source of error. In the UK for instance, Twitter users are more likely to be male and young compared to the national population (Sloan, 2017), and in the U.S. men and residents of densely populated areas are overrepresented (Hecht and Stephens, 2014), while there exists a mix of over- and undersampling on Twitter for users with specific racial backgrounds (Mislove et al., 2011). In general, however, the exact inclusion probabilities for a specific demographic group on any platform in a given country or region are unknown. And while not all social sensing tasks require representative data (An and Weber, 2015), recent studies have pointed out the fallacies in predicting phenomena via social sensing without controlling for sampling biases in social media data (Jungherr et al., 2012; Gayo-Avello, 2012, 2013), with research applications aiming to draw inferences for nation-wide target populations often relying on such attribute data. Consequently, a systematic methodology for estimating them is needed.

Survey analysis researchers have dealt with non-representative polls and non-response through sample re-weighting—with post-stratification being a well-known technique (Bethlehem and Keller, 1987; Holt and Smith, 1979). These methods make use of basic demographics attributes: most prominently age, gender and location (Park et al., 2004; Wang et al., 2015). Where demographic details are provided, post-stratification has been a valuable technique (Wang et al., 2015; Zagheni et al., 2017). However, most social media platforms do not provide demographic data about their users, making it difficult to know or correct for such biases. Although promising techniques for extracting demographic attributes from social media have been proposed, there exists no robust multilingual approach for this task. We address these challenges by inferring basic demographic attributes and correcting for selection biases on a large sample of multilingual profile data from Twitter (Figure 1), one of the most used platforms for social sensing given its ease for obtaining data.

This work offers the following three contributions. First, we introduce a new multilingual, multimodal, multi-attribute deep learning system for inferring demographics of users (§2). This pipeline is built from the ground up to enable inference in 32 different languages, addressing the need for such methods beyond English and outperforming state-of-the-art methods on the tasks of predicting the age, gender and is-organization state of Twitter users. We show this system reduces algorithmic bias with respect to skin tone over the best-performing commercial systems. Second, we formalize a statistical framework for models that debias non-representative samples (§3). Our models explicitly learn per-stratum inclusion probabilities from data, whereas typical post-stratification methods either assume their knowledge or focus on obtaining a post-stratified estimator of a response variable. Third, in real-world evaluations, we show that our debiasing models significantly reduce the error rate in comparison to a model without post-stratification on the inferred demographic attributes (§4). In these ways, we make a significant step towards representative social sensing in downstream applications using multilingual social media.

2. Demographic Inference

We propose a new demographic inference model for three attributes: gender, age, and a binary organization indicator (“is-organization”). The first two attributes, gender and age, are widely reported in census data and are core features to accurately measure demographic biases. The third attribute was selected based on a known confounder for people-based studies of online platforms: the presence of accounts belonging to organizations (Alzahrani et al., 2018; McCorriston et al., 2015), which shall be distinguished from individual accounts. Next, we describe the model, its data, and its evaluation.

2.1. Classification Task

We consider the task of estimating population count in Twitter given a stream of user tweets. This scenario is motivated by the common research use case for social sensing on Twitter, where a researcher consumes one of the Twitter streams (e.g., the 10% or 1% sample streams) and desires to make some analysis from the tweets of a broader phenomena that extends beyond Twitter. In such a scenario, many users are expected to appear rarely–possibly only once–which hinders the use of existing demographic methods that rely upon large amounts of text to infer gender (Tannen, 1991, 1993; Kendall et al., 1997; Coates, 1998; Eckert and McConnell-Ginet, 2003; Lakoff and Bucholtz, 2004; Coates, 2015; Chen et al., 2015; Zhang et al., 2016) and age (Rosenthal and McKeown, 2011; Goswami et al., 2009; Rao et al., 2010; Nguyen et al., 2011, 2013; Sap et al., 2014).

Instead, we avoid making inferences based on tweets and focus on using only the information associated with a user’s account, which allows our method to easily scale to large volumes of users without the need for significant quantities of text. Not stratifying users based on tweets has the added advantage of preventing downstream demographic biases to any social sensing tasks that analyze the same language. Ultimately, our method uses four sources of information, username, screen name, biography, and profile image, and is designed to operate effectively on multilingual data.

We formalize the three classification tasks as follows. Gender and is-organization are modeled as binary classification tasks.111We base our notion of gender as one of performance (DeFrancisco et al., 2013), in which individuals adapt their style, name, and picture to (de)emphasize certain aspects of their gender identity (Eckert, 2008). While our training data uses self-declared binary gender identities, the M3 model reports gender identity as a continuum using prediction probability along a graded scale. We recognize that the current approach does not yet support non-binary gender identities, which we view as an important future task for full representation. Age is known to be a difficult inference task in social media (Nguyen et al., 2013; Zhang et al., 2016). Here, we opt to categorize age in four levels: $\leq 18$, $(18,30)$, $[30,40)$, $[40,99)$. Our choice is motivated by (i) our ability to align these age ranges with census data reports and surveys (UK Office for National Statistics, 2015; United States Department of Education, 2009), and (ii) the difficulty of the task (even for humans) when finer-grained age classes are used (Dehon and Brédart, 2001; Van Rijsbergen et al., 2014), while (iii) still making the resulting classes amenable to downstream tasks. While other attributes are possible for use in debiasing (e.g., education, income, etc.), ground truth population statistics for these attributes are not widely available in censuses, making it difficult to train and evaluate any model.

2.2. Method and Training Procedure

Demographic attributes are often expressed in both image and text. We propose a new multimodal inference model with a novel training technique that leverages both of these sources to significantly augment our predictive accuracy. We first describe the model and each modality and then discuss the training procedure.

2.2.1. Model

The full model comprises two separate pipelines for processing a profile image and each of the three text sources of information, as shown in Figure 2. Here, we use a shared pipeline for all three attributes, using multi-task learning for the output. We refer to this as the M3 model, after its multimodal, multilingual, and multi-attribute abilities. Following, we describe the architectures of each modality and how they are combined.

Image Model The image classifier was constructed using DenseNet (Huang et al., 2017) based on initial performance experiments over current state-of-the-art vision models on our data. We scale all profile images to 224x224 to meet the input size of the model.

Text Model Separate character-based neural models were trained for each text input and then a multi-input model was tuned based on the weights from each single model. Embeddings for username and biography were limited to the 3000 most frequent characters in our training corpus, with remaining characters represented with an encoding for their unicode category; screen name are embedded in the ASCII range. We use 2-stack bidirectional character-level Long-short Term Memory (LSTM) architecture (Hochreiter and Schmidhuber, 1997) to capture necessary information. To incorporate potential different meanings in different languages for the same character, we concatenate a separate trainable language embedding to each character embedding and add a fully connected layer before being fed into an LSTM. The fusion of character-based input with language embeddings provides two critical benefits that let us scale to the linguistic diversity seen in global platforms like twitter. (1) Character-based RNNs are able to capture morphological regularity within a language and substantially reduce the parameter space due to the reduced vocabulary size of the model (i.e., the set of characters) as compared with word-based models with orders-of-magnitude larger vocabularies (Kim et al., 2016; Faruqui et al., 2016; Chung et al., 2016). (2) By fusing language embeddings with character embeddings, the joint embedding lets us represent shared linguistic regularity across languages, which lets us better scale to the multilingual environment.

Full Multimodal Model To construct the full model, separate models are fully trained for image and text for all attributes. Then, the softmax output layer of each is removed and the two models are joined into a new model with a modality drop-out layer (described next), two fully connected layers of size 2048, a Rectified Linear Unit (ReLU) layer (Nair and Hinton, 2010), and separate output layers for each task. This full model is then fine-tuned using the pretrained single-modality models’ parameters for initialization.

2.2.2. Training Procedure

Rather than training M3 end-to-end initially, we combine three procedures for the training process: co-training, multilingual data augmentation, and modality drop out.

Co-Training Both image and text can have clear signal of demographics. Given the potentially-limited training data for speakers of less common languages, we take advantage of the multiple views of the same users by incorporating co-training: a semi-supervised learning technique. Co-training first learns a separate classifier for each view using any labeled examples. The most confident predictions of each classifier on the unlabeled data are then used to iteratively construct additional labeled training data (Blum and Mitchell, 1998). Ardehaly and Culotta (Ardehaly and Culotta, 2017) recently showed co-training was effective for demographic classification in social media data using images and text in a graph-based co-training learning procedure.

In comparison to the variance of gender and age expressions in textual data across sociolinguistic contexts (Schler et al., 2006; Bamman et al., 2014), image data provides a more universal depiction, acting as an analog to a universal pivot language in machine translation (Wu and Wang, 2007). Thus, we train the image-based portion of the M3 model first, which identifies high confidence labeled data from an unlabeled multilingual dataset. Second, the text-based portion of the M3 model is trained using the original dataset plus the text data associated with any high-confidence image-based classifications. Finally, we train the full M3 model with the training dataset and all high-confidence predictions from the unlabeled data. All models are trained as multi-task, where a common input representation and architecture is used to predict each output tasks. The probabilities for each task were represented using a top softmax layer.

Multilingual Data Augmentation The M3 model is designed to operate in online environments on tens of languages. However, few approaches and little labeled data exists for this task for languages other than English, with some exceptions (Ciot et al., 2013; Goot et al., 2018; Rangel et al., 2018). Even with co-training, the relative differences in number of speakers between languages (e.g., English vs. Slovenian) create a substantial class imbalance which would prevent the model from effectively learning signals of each attribute in users’ biographies written in less-frequent languages. Therefore, we perform data augmentation by using automatic machine translation of the training data.

Given the scale of our training data (described later in §2.3), the use of online translation services is prohibitive both in terms of time and cost. Instead, we opt to use word-based translation using the MUSE bilingual dictionaries (Conneau et al., 2017). For each instance in the training data, all words in the bilingual dictionary are replaced with the translation. A translated instance is only kept if 80% or more of its words could be translated. While word substitution typically produces translations that are more incorrect and less fluent, we observed that the remaining biography translations were often of reasonable quality due to the declarative nature of biographies, which often consist primarily of lists of readily-translatable noun descriptions or attributes of the user without using full grammatical sentences. Such translations also largely capture topical and semantic regularity, even if the morphology is not precise.

Modality Drop-out The co-training model uses a conjoining architecture with pre-trained weights from the image model and the multi-input text model. To facilitate the model using each source of information, we introduce a new technique, modality dropout, that can fully hide a source of information (e.g., the biography) during training. We add an extra input dropout with a probability of 0.25 (i.e., keep 3 out of 4 input sources on average) to better regularize the model and support use cases where not all 4 input sources are available. When doing dropout, images were replaced by a random matrix while text was turned to a specific empty embedding.

2.3. Data

Training and evaluating M3 uses five distinct datasets: (1) a large dataset of Twitter profiles whose gender and age are heuristically identified, predominantly in English, (2) a curated dataset of Twitter accounts belonging to organizations, (3) an image dataset of faces from IMDB and Wikipedia, (4) a massive unlabeled set of users, and (5) a crowdsourced dataset for all three attributes that spans all 32 languages in our study. We describe each next.

2.3.1. Heuristically-identified users

Heuristically-identified users were drawn from a 10% sample of Twitter from 2014 to 2017 on Twitter, where roughly 40% of the users are English speakers (Hale, 2014). Here, we identify gender and age from fixed expression in users’ biographies that signal age or gender, e.g., “mother of two wonderful kids” or “26 y/o dude” using a gender- or age-signaling word. Age data was further augmented by identifying tweets wishing another user a happy birthday (cf. Al Zamal et al. (2012)). These pattern-based approaches are known to be highly precise for certain attributes when properly constructed (Beller et al., 2014; Bergsma and Van Durme, 2013). Patterns were created for five languages: English, Spanish, French, German, and Swedish.222Languages were selected to capture linguistic diversity and where high precision demographic-identify patterns could be created easily. Organization accounts were manually curated by identifying 676 Twitter lists that primarily contained organizations, such as non-profits, local clubs, companies, or municipal services. These lists resulted in 59.92K unique organization accounts. For heuristically-identified users and organization accounts, we collect their current profile image, screen name, username, and biography. User age is adjusted to the present day. For heuristically-identified users, the biography is altered to replace the gender- or age-indicating word with a special token so that the model is forced to look for additional cues to recognize each attribute. The IMDB-WIKI dataset (Rothe et al., 2016) consists of 523,051 images from headshots of actors in the IMDB and profile pictures in Wikipedia. This dataset provides an auxiliary source of information for fine-tuning the image-based part of the M3 model.

These three datasets constitute our initial data prior to performing the three-step annotation procedure. Table 1 shows the detailed statistics of the data for each step.

Co-training depends on having access to a large set of unlabeled users whose high-confidence labels from one modality (e.g., profile image) can be used to augment the training set for another modality. Here, we collect 36.97M profiles speaking one of the 32 languages commonly spoken in Europe but for which we have no groud truth label. Languages were identified using CLD2 (McCandless, 2010). Note that users in this unlabeled dataset will only become training instances if one view’s classifier (text or image) labels them with high confidence.

2.3.2. Crowdsourcing Data

Current gender, age, and organization datasets are nearly all produced for English-speaking users, which prevents us from evaluating the M3 model in the multilingual environment. Therefore, we constructed a new dataset of up to 200 randomly-selected Twitter users speaking one of each of the 32 languages in our study. Three annotators on Figure Eight333https://www.figure-eight.com/ were shown the usernames, screen names, short biographies, and profile images of each Twitter account. We instructed workers to determine gender and age of a given user from two drop-down menus based only on this information. The selection options regarding gender also included a category for profiles belonging to multiple persons, and to a non-person (e.g., organization/bot). Regarding age, annotators were given seven categories to choose from: $<=18$, $(18,30)$, $[30,40)$, $[40,50)$, $[50,60)$, $[60,70)$, $>=69$. They could select “don’t know” for all questions, if they felt not enough information was available. We employed Figure Eight’s selection mechanism for speakers of the target language of the set to be annotated, or chose workers by their country of residence, if one of that country’s main spoken languages was identical with the target language of the task. Additionally, each job included a set of 30 test questions, annotated as a gold standard by the authors. These were used to screen for spam but also for any non-speakers of the target language remaining after our first filter; to this end, at least 10 questions were only solvable by sufficiently understanding the target language.

Annotation reliability was measured for each Figure Eight job using Krippendorff’s $\alpha$ (Krippendorff, 2011). For the computations of $\alpha$, we excluded all English test-questions from the 31 non-English jobs, profiles with only one annotation, and included “don’t know” answers as an answer option. To establish a measure of how well workers did on the age classification in respect to the age brackets classified by M3, we collapsed age bracket encodings over 40 into one category. Across all languages, the mean $\alpha$ for the gender annotations lies at 0.81 and at 0.57 regarding the age brackets, hinting at the high difficulty of the Twitter profile age classification task for humans. For a separate measure, we recoded all gender and organization annotations into the two categories “is organization” and “is not organization” and calculated Krippendorff’s $\alpha$ based on these data, resulting in a mean of approximately 0.75 across all languages, showing that workers could identify non-personal accounts reasonably well. The final dataset is constructed from all instances where at least two annotators agreed on the label for a particular attribute (majority vote), resulting in 4,732 profiles across 32 languages.

2.3.3. Data Partitions

All self-reported labeled data is partitioned into 80% train, 10% development, and 10% test splits. No official train/test/dev splits are available for the IMBD-WIKI datasets so we perform our own partition with the same percentages as for self-reported. During model training, only the development set for self-reporting users are used for model selection. Owing to its small size, crowdsourcing data was only used for testing purposes. For the organizational data, we create a full dataset with non-organizational accounts by randomly sampling accounts from the heuristically labeled data to attain a 1:9 ratio, following the report in McCorriston et al. (McCorriston et al., 2015) that organizations make up 10% of the accounts on Twitter.

2.3.4. Co-Training Setup

The training data is balanced for each stage, where we oversampled gender and organization status, and undersampled the two most-frequent classes (0-18 and 19-29) while oversampling the other two age classes. High confidence thresholds were set to 0.9 for gender and organization status in both the image and text modalities. As a four-class classification task, the high-confidence for age is set to 0.7 in the image classifier, based on early examination of the prediction quality. We observed that age is more readily predictable in text and therefore used a threshold of 0.9 to keep quality high while also providing a sufficient number of new accounts for co-training.

Separate image and text models are each trained for 10 epochs. Due to the imbalanced number of instances per attribute (Table 1), each epoch contains 10000 steps where each step contains a mini-batch of 128 instances balanced across the three attributes. This setup ensures the model sees instances of the smaller datasets (age and organization). The image model is trained first, and its high-confidence instances are then used to train the text model. The final co-training model was trained for 5 epochs with input dropout and another 3 epochs without input dropout for better convergence. Amsgrad optimizers (Reddi et al., 2018) were used in all training process, with a learning rate of 0.001 for ground-up models and 0.0005 for fine-tuning models. Text models were trained on one GPU, while the image and co-training models were trained in parallel on 4 GPUs. The full pipeline was built in PyTorch (Paszke et al., 2017).

2.4. Evaluations

Accurate inference of representative attitudes from social media relies upon accurate stratification approaches. Here, we test the M3 model against current state of the art systems to describe the potential for error to affect demographic bias.

2.4.1. Comparison Systems and Data

The M3 model is compared with state-of-the-art systems from each modality and attribute. For images, we compare with Face++ (Jung et al., 2017) and Microsoft Face API (Azure, 2018) on age and gender performance. For text, we compare with three current state-of-art systems for inferring gender from usernames, genderperformr (Wang and Jurgens, 2018), demographer (Knowles et al., 2016) and Jaech et al. (Jaech and Ostendorf, 2015), which could feasibly operate on names from multiple languages.

No multilingual organization recognition system exists; so, we limit our evaluation to the only publicly-available dataset of person-vs-organization which was scored by the Humanizr (McCorriston et al., 2015) and Demographer (Wood-Doughty et al., 2018) systems. This data consists of a uniform sample across Twitter accounts, which were approximately 10% organizations. Here, we recollect the user profiles for their 20,273 accounts, of which 18,587 (91.6%) were still available as of October 2018, which matched the distribution originally reported. Their model was evaluated on the dataset using cross-validation, whereas we treat the entire data as a test set and report performance.

2.4.2. Gender Recognition

M3 produces state-of-the-art gender recognition for each of the three image-based datasets (IMDB, WIKI, and Twitter), as seen in Figure 3. M3 provides significant improvements in performance (F1) at each comparison system’s level of recall.444The coverage reported is the maximum available from the commercial APIs, which did not find a face or did not provide a gender/age estimate for each photo. The one exception to this trend is the performance of the MSFT classifier on the WIKI dataset, which has highly similar performance to M3. Given the MSFT’s model large jump in performance, we hypothesize that their model may have been trained on Wikipedia data, although this cannot be confirmed. We observed that the increase in coverage of M3 is due in part to the model learning non-facial attributes associated with more than one gender, e.g., certain sports with the male gender, which allow M3 to process profile photos that other models cannot.

On the text-based data, M3 outperforms all systems except one when using just the username and masking all other information, shown in Table 2. When the model is allowed to see all other text information, M3 performance improves substantially, indicating it can effectively fuse several sources of information. While this latter setup uses more information than comparison systems, in practice, the Twitter API includes all text information used by M3 by default; so the only additional step is downloading the profile image.

2.4.3. Age Recognition

Age recognition is a significantly more difficult task in social media, as seen by the performances in Figure 4. The M3 model offers similar performance to commercial models on the IMDB and WIKI datasets, which primarily feature face-forward headshots in good lighting. However, in real-world Twitter profiles (shown in blue), the M3 model substantially outperforms both in F1 at each of their respective coverage levels, with 0.16 and 0.11 absolute improvements over Face++ and Microsoft, respectively.

2.4.4. Organization Recognition

For any task that draws inferences about human actors, non-human and specifically organizational accounts are a major source of error. The results of Table 3 show that our model is able to identify these accounts with 16.3% higher accuracy than the next closest system with no drop in performance at recognizing humans. Results on our test set from our organization data (§2.3) in Table 4 (top right) show similar performance, with an overall F1 of 0.898 that indicates high performance for both classes.

2.4.5. Multilingual Performance

As a full test of M3, we evaluate it against our crowdsourced account labels in 32 languages, shown in Table 4 top right. Performance in this multilingual setting is on par with the performance on the primarily-English heuristically-labeled data, though age classification remains the most difficult task. Figure 5 shows the performance per language for each attribute. Performance at predicting gender and is-organization are similar for most languages, while age performance varies significantly by language from 0.28 F1 for Bosnian to 0.73 for Slovenian and Welsh. These results together indicate that M3 is sufficiently accurate in the multilingual European environment.

2.4.6. Ablation Study

To examine which parts of the system are contributing to performance, we perform an ablation study by (i) restricting the model to one modality and (ii) training a model without using co-training or translation for data augmentation. We test each model on (i) the heuristically-labeled and organisation data and (ii) the multilingual data, which is representative of the performance on the data used in our downstream task.

The results shown in Table 4 reveal two main trends. First, the model benefits from both modalities. The removal of textual information causes the biggest performance drop. Second, although co-training and translation hurt performance in the heuristically-labeled data, they produced a substantially better model when evaluated on multilingual data. Since the heuristically-labeled data is primarily in English, translation potentially adds noise and forces the model to represent a larger space of inputs that are not represented in the test data. This performance difference indicates that the two data augmentation techniques are highly beneficial when bootstrapping a model from mostly monolingual data.

2.4.7. Test for Algorithmic Bias

Image-based models for gender recognition are known to suffer from algorithmic bias in recognizing darker-skinned individuals (Buolamwini and Gebru, 2018). This bias is thought to be a result of non-representative training data the underreprentation of darker skin tones in the training data. As our focus is on European countries, such a bias could be present in the M3 data. However, because our co-training procedure uses a large unlabeled set of users from across the globe, these users can potentially provide a more representative sample and reduce algorithmic disparity. To test for algorithmic bias, we compare the M3 model’s performance on the Gender Shades dataset (Buolamwini and Gebru, 2018), which contains 3964 gender-annotated facial images balanced across light and dark skin tones. In this earlier algorithmic audit, the Microsoft (MSFT) and Face++ gender classifiers performed substantially worse on women, especially darker skinned women.

The gender inference performance, shown in Table 5, reveals that M3 has substantially less algorithmic bias relative to the two best-performing commercial systems also tested on Gender Shades. Our model significantly improves performance on dark-skinned women in comparison to the other two systems which had the largest performance disparity on that demographic. However, M3 is least accurate on darker-skinned males, which indicates that additional work is needed to reach performance parity in skin tones. Nevertheless, this analysis also indicates that M3 is more suitable than existing systems for operating in the global environment and increases the robustness of downstream social sensing applications.

3. Learning inclusion probabilities

In social sensing studies, the results of measurements performed on a given platform, e.g., a social media site, are studied to understand the behavior of a population. Often, such measurements are biased (Ruths and Pfeffer, 2014), as individuals with certain demographics are more likely to join these platforms, e.g., young people may be more likely to join Twitter. Obtaining representative estimates in these scenarios is challenging, as the probability of an individual with given demographics to be on a given platform, also referred to as inclusion probability in sampling methodology, is typically unknown.

Here, we estimate these inclusion probabilities as a function of demographics. To this end, we learn the debiasing coefficients on the grounds of statistical survey analysis with missing data (Bethlehem and Keller, 1987; Little and Rubin, 2002; Särndal and Lundström, 2005). Specifically, we derive estimators for the numbers of individuals by making an assumption that per-strata inclusion probabilities are equal for individuals within a group (e.g., a country) and different between groups (e.g., between countries). This assumption allows us to get both across-groups global debiasing coefficients (i.e., the biases of Twitter users) and the group-specific debiasing coefficients (i.e., per-country demographics on Twitter, e.g., developed countries may have more and diverse users on Twitter). Our approach learns the inclusion probabilities, while typical post-stratification methods either assume these probabilities are known or focus on obtaining a post-stratified estimator of a response variable.

3.1. Formulation of Debiasing Models

Consider a population $U$ of $N=|U|$ individuals with certain demographics. For simplicity, we focus on the case of only two discrete demographic variables, say age $a$ and gender $g$, which are distributed following $P_{\text{N}}(a,g)$, but the following reasoning applies to other demographics as well. Out of these $N$ individuals, say, $M\leq N$ joined a certain online platform with a probability depending on their demographics. For instance, individuals joined Twitter and younger people were more likely to join than older ones.

In survey analysis, the probability that an individual with certain demographics joined a certain platform corresponds to the inclusion probability of the stratum representing these demographics (Bethlehem and Keller, 1987; Särndal and Lundström, 2005). In the simplest scenario, when the probability of joining the platform is homogeneous in time and across individuals, this probability can be expressed as the ratio of the number of Twitter users with the demographics $a$ and $g$ to all individuals, i.e., $\pi(a,g)=\frac{M(a,g)}{N(a,g)}=\frac{MP_{\text{M}}(a,g)}{NP_{\text{N}}(a,g)}$, where $P_{\text{M}}(a,g)$ and $P_{\text{N}}(a,g)$ are the distributions of demographics of Twitter users and the overall population, respectively. However, the inclusion probability may vary between individuals. To account for this, we discuss the homogeneity of inclusion probability for a given partition of the population. A partition of the population $U$ is defined through non-overlapping and non-empty subsets $U_{i}$ of the population that together constitute $U$, i.e., $\bigcup_{i}U_{i}=U$. Typically such subsets will have a certain meaning, e.g., a natural partition of a population is a split by countries, regions, or cities. The total number of individuals in the subset $i$ is $N^{i}$ and the number of individuals having particular demographics is $N^{i}(a,g)$.

3.1.1. Homogeneous bias

The inclusion probability, which governs the bias, is homogeneous with respect to a given set of demographics and a given partition of the population, if and only if $\pi^{i}(a,g)=\pi(a,g)$ for each subset $i$ of the partition, i.e., the inclusion probability does not depend on the elements of the partition.

If the inclusion probability is homogeneous, then we can write

[TABLE]

Thus, to obtain the inclusion probabilities, we regress $N^{i}$ against $M^{i}(a,g)$. If the condition of homogeneity holds, then the regression coefficients are equal to $1/\pi(a,g)$.

3.1.2. Inhomogeneous bias

If the inclusion probability is inhomogenous for given demographics and partition, then we shall model each subset of the partition separately or use a different model that relaxes the homogenity assumption by specifying the functional form of inhomogeneity. Here, we consider the inhomogeneity of the form

[TABLE]

where $\nu$ is an unknown exponent and $f_{1}$ and $f_{2}$ are unknown functions.555Note that the aforementioned homogeneity assumption is a special case of this assumption, i.e., this assumption relaxes the homogeneity assumption. Under this assumption, $\log N^{i}(a,g)=\log\frac{M^{i}(a,g)}{\pi^{i}(a,g)}=(1-\nu)\log M^{i}(a,g)-\log f_{1}(a)-\log f_{2}(g).$ This time, to obtain the debiasing coefficients and corresponding inclusion probabilities, we regress the ground-truth population, $\log N^{i}(a,g)$, against the biased measurements, $\log M^{i}(a,g)$, and the demographic indicator variables, i.e.,

[TABLE]

where $\delta_{ij}$ is Kronecker delta, i.e., $\delta_{ij}=0$ if $i\neq j$ and $\delta_{ij}=1$ if $i=j$, $\beta_{1}=(1-v)$, $\beta_{a}=-\log\ f_{1}(a)$, and $\beta_{g}=-\log\ f_{2}(g)$. From these regression coefficients we can obtain the inclusion probability $\pi^{i}(a,g)$ of a set of samples $i$ with the demographics $a$ and $g$ via Equation 2. This debiasing model has been proposed recently by Zagheni et al. to predict the number of migrants based on Facebook data (Zagheni et al., 2017), but its derivation and explicit formal interpretation have not been provided until now, to the best of our knowledge. Note that this estimation method of inclusion probabilities requires the ground-truth joint counts $N^{i}(a,g)$ at the time of training, whereas the model based on the homogeneity assumption requires only the total counts $N^{i}$. The availability of the joint counts is often limited in practice because of insufficient number of samples per stratum.

3.2. Discussion

We derived debiasing models by making homogeneity or inhomogeneity assumptions. These models predict population size, so they can be evaluated by measuring the error of predictions in cross-validation settings or via model selection methods. In this way, we test which assumption is closer to reality and find the most accurate method for the given dataset. We evaluate these models in the next section using the population of regions in EU countries.

4. European Population Inference from Twitter data

Here, we use the debiasing models formulated in the previous section to obtain the debiasing coefficients and the corresponding inclusion probabilities for Twitter users of different countries. To this end, we regress the country-level ground-truth number of people living in a certain location against the number of Twitter users of different demographics.

4.1. Debiasing Models for Population Inference

More specifically, we evaluate five models requiring different amounts of data. The first model is a baseline, the next three models are based on the assumption of homogeneous inclusion probabilities (Equation 1), whereas the last one is based on the inhomogeneity assumption (Equation 3):

$\mathbf{N\sim M}$**: **

is our base model that uses only the total population count from the census ($N$) and Twitter ($M$).

$\mathbf{N\sim\sum_{g}M(g)}$**: **

uses gender marginal counts only (i.e., the total counts of males and females not broken down by ages).

$\mathbf{N\sim\sum_{a}M(a)}$**: **

uses age marginal counts only.

$\mathbf{N\sim\sum_{a,g}M(a,g)}$**: **

uses the joint histograms inferred from Twitter but only the total population values from the census.

$\mathbf{\text{{log}}~{}N(a,g)\sim\text{{log}}~{}M(a,g)+a+g}$**: **

uses the joint histograms inferred from Twitter and the joint histograms from the census.

Note that Twitter users are biased in various ways: the platform is more accessible to tech-savvy individuals and citizens of various countries use it to a different extent (Hale, 2014). To distinguish the global effect of Twitter on the overall bias from the local effect of country, we use multilevel models. Namely, all slopes and intercepts in the introduced models have random effects specific to a country. Thus, from fixed effects of the model, we obtain global debiasing coefficients (i.e., the biases of Twitter users) and, from random effects, the country-specific debiasing coefficients (i.e., a given country has its own bias towards Twitter).

Note that the homogeneity and inhomogeneity assumptions apply within each country separately. Namely, we group samples by regions of a country, described next. Then, the homogeneity assumption translates into the same inclusion probability for a given demographic across all regions of a country, whereas the inhomogeneity assumption translates into a dependence of the inclusion probability on regions of a country that follows Equation 2.

4.2. Data

We retrieved joint population distributions for age and gender on a regional level (NUTS3) for 26 countries of the European Union plus four EFTA members as made available through the CensusHub system of Eurostat, the European Statistical Office.666https://ec.europa.eu/CensusHub2/: Luxembourg and Belgium were not available. Additional EFTA countries: Switzerland, Iceland, Liechtenstein and Norway. All census data is from 2011, the most recent year for which comprehensive census data was collectively reported to Eurostat. NUTS3 regions are the finest-grained level of the “Nomenclature of Territorial Units for Statistics” used as a standard for statistical reporting in EU member states (European Commission, [n. d.]). They are based on existing local administrative boundaries and are usually at the level of local districts. Although sizes differ to a certain extent per country, NUTS3 provide the most standardized cross-country geographical units to subdivide populations. We use the 2010 iteration of the NUTS3 regions as these correspond with the 2011 census subdivisons and is the most recent census available.

Our sample of social media users is derived from the random 10% stream of tweets from Twitter. We recorded all users observed from September 2015 to January 2016 and downloaded profile photos for all users in spring 2018. The location of each user is inferred using the method of Compton et al. (Compton et al., 2014). This model predicts a user’s latitude and longitude and was shown to be least biased with respect to urban and rural areas (Johnson et al., 2017), which is important given the diversity in population centers in our study. The model was trained on a social network of 781M edges constructed from Twitter data spanning 2012 to 2017. Five-fold cross validation reports a median inference error of 7.9km, which is sufficiently accurate for the geographic granularity we use here. Each user is assigned to a NUTS3 region in Europe if the inferred latitude-longitude pair of that user lies within the boundaries of the NUTS3 regions; users not in these regions are discarded. Ultimately, we obtain a dataset of 3,202,964 users within the NUTS3 regions for our study. In the remainder, we will refer to NUTS3 regions simply as “regions.”

The age, gender, and is-organization variables for each user in our dataset are inferred using M3. As an additional experiment to quantify the impact of including non-human accounts, we ignore any organization classification and treat all accounts as humans, grouping them according to their inferred demographics.

4.3. Results

We evaluate the debiasing models in the following cross-validation settings: leave one region out, leave one country out (i.e., leave out all regions from a given country), and leave one stratum out (e.g., leave out only males aged 30-39). Per case, we measure the mean absolute percentage error of the population estimates for the left-out samples, i.e., $\text{MAPE}(N)=\frac{100\%}{n}\sum_{i}\frac{|\hat{N}_{i}-N_{i}|}{|N_{i}|}$, where $\hat{N}_{i}$ and $N_{i}$ are the predicted and actual population sizes, respectively, and the sum is over all regions. Note that the model $\log N(a,g)\sim\log M(a,g)+a+g$ operates naturally in the log space; hence, before calculating the error of this model we first exponentiate the predicted population sizes to make the results more comparable across all models. Its results should further be compared to the remaining models with care, as the effects of moving to log space cannot be clearly untangled from gains due to learning on joint age and gender attributes.

The results of the leave-one-region-out evaluation show a clear benefit to debiasing based on the inferred demographics (Figure 6). For the $N\sim M$ model without demographics, MAPE is 88%. The inclusion of the inferred gender or age demographics in the debiasing models, via $N\sim\sum M(a)$ and $N\sim\sum M(g)$, decreases MAPE to 59% and 61%, respectively. Including inferred joint distributions, e.g., the number of males of age 30-39, in the model $N\sim\sum M(a,g)$, decreases MAPE further to 54%, even though we do not use joint-distribution data from the census for training. In fact, the use of census joint distributions at the stage of model training along with a move to log space, via $\log N(a,g)\sim\log M(a,g)+a+g$, further improves the accuracy, bringing the MAPE down to 33%. To understand the effect of accounts belonging to organizations on this population prediction task, we compare the results of the debiasing models with and without removing the organizations (Figure 6). Removing organizations results in a small reduction in error for all but the model trained with only marginal gender counts, though these differences are not statistically significant. A further analysis shows that the presence of organizations is heavily skewed towards populous metropolitan cities and most regions have very few organizations (i.e., organizations are not equally geographically distributed). However, for populous regions, organizations can create significant error.

To gain further insights into the introduced models, we show scatter plots of true and predicted population sizes for each model (Figure 7). The model with joint Twitter and census counts is noticeably closer to the $y=x$ line, likely because this models is trained in log space. For this model, we also depict the geographical variation of MAPE across the EU regions (Figure 8).

The above results test predictions when one region is left out and its population is predicted. However, in different circumstances, more or less census information could be available. We test two other cross-validation settings: leave one stratum out (e.g., leave out only females aged 30–39) and leave one country out (i.e., leave out all regions from a given country). The latter reflects the generalizability of the model to completely unseen countries where platform adoption probabilities and country-specific biases are not known. The evaluation results for all three cross-validation settings are plotted in Figure 9 for the most accurate model, i.e., $\log N(a,g)\sim\log M(a,g)+a+g$. From the first additional cross-validation setting, we learn that hiding the population sizes for a specific stratum in all regions results in a minimal penalty to the prediction accuracy. For the second additional cross-validation setting, we see that the error rate nearly doubles with an average MAPE of 81%. This result suggests that knowledge of country-specific platform biases is important for accurate estimates and that at least some regions within the country should be seen during a model’s training time to reach high performance.

4.4. Discussion of Debiasing Results and Potential Sources of Error

Debiasing social media data samples is a difficult task, but our results show that automatic post-stratification with respect to inferred age and gender notably improves population estimation results. Furthermore, our predictive models are fully interpretable, which allows us to estimate the inclusion probabilities and share them with the research community for future reuse.

Even when joint distributions are not given by the census, inferring joint distributions from social media data with the model $N\sim M(a,g)$ provides a significant increase in accuracy in population prediction tasks compared to the baseline without debiasing. However, the model $\log N(a,g)\sim\log M(a,g)+a+g$ is notably more accurate, suggesting that the assumption of homogeneity does not hold for the partition of citizens of a particular country, gender, and age into regions. Note, however, that the predictions of the most accurate model are not perfect. This may be caused by the time mismatch of about 5 years between the Twitter and NUTS3 datasets. On the other hand, this point opens the door for developing debiasing models based on other inhomogeneity assumptions and searching better partitions, e.g., our multilevel model could have more levels to capture biases shared by smaller regions within a country.

To learn more about potential confounders, we compare the MAPE of the model $\log N(a,g)\sim\log M(a,g)+a+g$ to three variables: the area of the region, its population density, and average income (Figure 10). NUTS3 regions are defined by each country and vary considerably in land area from city-states in Germany to large regions in Sweden (up to 98,000 km2). We do not find correlations of land area with the MAPE error. Hecht and Stephens (Hecht and Stephens, 2014) found a clear urban–rural bias in social media data: there tend to be more users, more information, and higher quality information per capita within metropolitan areas. Despite this bias, we find our estimates not heavily correlated with population density, suggesting our models do well at debiasing these differences in inclusion probabilities. Only two countries (Czech Republic and Norway) show a correlation that is statically different from zero across our models. Finally, income has been suggested as an important explanatory variable for who uses Internet-based platforms and hence could be a potential confound to our estimates (Chinn and Fairlie, 2007; Jansen, 2010). However, we similarly find that income and MAPE are not correlated significantly in our models.

5. Conclusion

The everyday opinions expressed in social media provide promising opportunities for measuring population-level statistics for health metrics, political outcomes, or general attitudes. However, social media is a non-representative sample of the population due to demographic skew in usage frequencies and access rates. As such, any direct estimate from a platform is likely biased towards certain demographics. This work provides a holistic solution to this problem by developing a novel method for assigning users to demographic strata and exploiting it in a regression framework for debiasing that allows direct estimation of the probability of an individual with given demographics to be on the given social media platform. Our work provides three main contributions. First, we introduce a state-of-the-art neural system for multi-attribute classification in 32 languages. This contribution also includes the system release and the creation of a new dataset of gender, age, and is-organization annotations in 32 languages. Second, we derive a series of models to debias social media measurements and provide their explicit formal interpretations. Third, in a massive study of all of Europe, we show that our two methods are able to infer regional population counts accurately and provide demographic corrections for all downstream measurements on the grounds of the estimated inclusion probabilities. These results pave the way for more accurate social sensing by laying a foundation of representative population sampling in social media. Code, software, and debiasing coefficients pertaining to this work are released for public use at https://github.com/euagendas/.

Acknowledgements

This research has received funding through the Volkswagen Foundation and was supported by The Alan Turing Institute under EPSRC grant EP/N510129/1.