Toward Informal Language Processing: Knowledge of Slang in Large Language Models

TL;DR

This paper introduces a new benchmark dataset for evaluating large language models on slang processing tasks, demonstrating how it improves model performance and provides interpretive insights into informal language understanding.

Contribution

It creates a comprehensive, publicly accessible slang dataset from movie subtitles for evaluation and finetuning of LLMs, filling a gap in informal language processing research.

Findings

LLMs like GPT-4 perform well in zero-shot slang detection.

Smaller BERT-like models achieve comparable results after finetuning.

Finetuning GPT-3.5 with the dataset significantly improves performance.

Abstract

Recent advancement in large language models (LLMs) has offered a strong potential for natural language systems to process informal language. A representative form of informal language is slang, used commonly in daily conversations and online social media. To date, slang has not been comprehensively evaluated in LLMs due partly to the absence of a carefully designed and publicly accessible benchmark. Using movie subtitles, we construct a dataset that supports evaluation on a diverse set of tasks pertaining to automatic processing of slang. For both evaluation and finetuning, we show the effectiveness of our dataset on two core applications: 1) slang detection, and 2) identification of regional and historical sources of slang from natural sentences. We also show how our dataset can be used to probe the output distributions of LLMs for interpretive insights. We find that while LLMs such as GPT-4 achieve good performance in a zero-shot setting, smaller BERT-like models finetuned on our dataset achieve comparable performance. Furthermore, we show that our dataset enables finetuning of LLMs such as GPT-3.5 that achieve substantially better performance than strong zero-shot baselines. Our work offers a comprehensive evaluation and a high-quality benchmark on English slang based on the OpenSubtitles corpus, serving both as a publicly accessible resource and a platform for applying tools for informal language processing.