Human Language Modeling

TL;DR

This paper introduces human language modeling (HuLM), a hierarchical approach that captures human states influencing language, using a large transformer model trained on social media data, improving performance on multiple NLP tasks.

Contribution

The paper presents HuLM, a hierarchical human-level language model, and HaRT, a large transformer trained on social media data, advancing social media language understanding.

Findings

HuLM improves language modeling perplexity on social media data.

HaRT achieves state-of-the-art results on multiple downstream tasks.

Hierarchical modeling captures human states influencing language.

Abstract

Natural language is generated by people, yet traditional language modeling views words or documents as if generated independently. Here, we propose human language modeling (HuLM), a hierarchical extension to the language modeling problem whereby a human-level exists to connect sequences of documents (e.g. social media messages) and capture the notion that human language is moderated by changing human states. We introduce, HaRT, a large-scale transformer model for the HuLM task, pre-trained on approximately 100,000 social media users, and demonstrate its effectiveness in terms of both language modeling (perplexity) for social media and fine-tuning for 4 downstream tasks spanning document- and user-levels: stance detection, sentiment classification, age estimation, and personality assessment. Results on all tasks meet or surpass the current state-of-the-art.