Time Will Change Things: An Empirical Study on Dynamic Language Understanding in Social Media Classification

TL;DR

This paper empirically investigates how social media language evolution impacts natural language understanding models, highlighting the challenges of dynamic environments and evaluating unsupervised adaptation methods for improved robustness.

Contribution

It introduces a dynamic evaluation setup for social media NLU and assesses the effectiveness of combined unsupervised domain adaptation techniques in handling language evolution.

Findings

Language features evolve over time, reducing model accuracy.

Unsupervised adaptation methods improve robustness to language change.

Auto-encoding and pseudo-labeling together outperform individual methods.

Abstract

Language features are ever-evolving in the real-world social media environment. Many trained models in natural language understanding (NLU), ineffective in semantic inference for unseen features, might consequently struggle with the deteriorating performance in dynamicity. To address this challenge, we empirically study social media NLU in a dynamic setup, where models are trained on the past data and test on the future. It better reflects the realistic practice compared to the commonly-adopted static setup of random data split. To further analyze model adaption to the dynamicity, we explore the usefulness of leveraging some unlabeled data created after a model is trained. The performance of unsupervised domain adaption baselines based on auto-encoding and pseudo-labeling and a joint framework coupling them both are examined in the experiments. Substantial results on four social media tasks imply the universally negative effects of evolving environments over classification accuracy, while auto-encoding and pseudo-labeling collaboratively show the best robustness in dynamicity.