Lithium NLP: A System for Rich Information Extraction from Noisy User Generated Text on Social Media

TL;DR

Lithium NLP is a high-throughput, language-agnostic system designed to extract rich information such as entities, topics, hashtags, and sentiment from noisy social media text, outperforming some existing systems.

Contribution

The paper introduces Lithium NLP, a resource-efficient, scalable system capable of extracting diverse information from noisy social media data, with competitive performance.

Findings

Lithium NLP matches or exceeds state-of-the-art commercial NLP systems.

Supports multiple languages and high throughput processing.

Effectively extracts entities, topics, hashtags, and sentiment.

Abstract

In this paper, we describe the Lithium Natural Language Processing (NLP) system - a resource-constrained, high- throughput and language-agnostic system for information extraction from noisy user generated text on social media. Lithium NLP extracts a rich set of information including entities, topics, hashtags and sentiment from text. We discuss several real world applications of the system currently incorporated in Lithium products. We also compare our system with existing commercial and academic NLP systems in terms of performance, information extracted and languages supported. We show that Lithium NLP is at par with and in some cases, outperforms state- of-the-art commercial NLP systems.