Performance Comparison of Crowdworkers and NLP Tools on Named-Entity Recognition and Sentiment Analysis of Political Tweets

TL;DR

This study compares the accuracy of crowdworkers and NLP tools in named-entity recognition and sentiment analysis on political tweets, revealing that NLP tools can match or approach crowdworker performance in certain tasks.

Contribution

It provides a direct comparison of crowdworkers and NLP tools on political tweet data, highlighting the strengths and limitations of current NLP systems for these tasks.

Findings

Google Cloud NL matches crowdworker accuracy in NER

TensiStrength lags behind crowdworkers in sentiment analysis

NLP tools show potential but still have significant gaps in accuracy

Abstract

We report results of a comparison of the accuracy of crowdworkers and seven Natural Language Processing (NLP) toolkits in solving two important NLP tasks, named-entity recognition (NER) and entity-level sentiment (ELS) analysis. We here focus on a challenging dataset, 1,000 political tweets that were collected during the U.S. presidential primary election in February 2016. Each tweet refers to at least one of four presidential candidates, i.e., four named entities. The groundtruth, established by experts in political communication, has entity-level sentiment information for each candidate mentioned in the tweet. We tested several commercial and open-source tools. Our experiments show that, for our dataset of political tweets, the most accurate NER system, Google Cloud NL, performed almost on par with crowdworkers, but the most accurate ELS analysis system, TensiStrength, did not match the accuracy of crowdworkers by a large margin of more than 30 percent points.