Harnessing the Intrinsic Knowledge of Pretrained Language Models for Challenging Text Classification Settings

TL;DR

This paper explores leveraging pretrained language models' intrinsic knowledge to improve challenging text classification tasks, including distractor selection, generalization to unseen labels, and prompt sensitivity, advancing NLP capabilities.

Contribution

It introduces novel methods for distractor generation, label generalization, and prompt selection that significantly enhance performance in difficult text classification scenarios.

Findings

Models outperform human accuracy in distractor selection.

Improved generalization to unseen labels with domain-independent descriptions.

Enhanced prompt effectiveness reduces model sensitivity and ambiguity.

Abstract

Text classification is crucial for applications such as sentiment analysis and toxic text filtering, but it still faces challenges due to the complexity and ambiguity of natural language. Recent advancements in deep learning, particularly transformer architectures and large-scale pretraining, have achieved inspiring success in NLP fields. Building on these advancements, this thesis explores three challenging settings in text classification by leveraging the intrinsic knowledge of pretrained language models (PLMs). Firstly, to address the challenge of selecting misleading yet incorrect distractors for cloze questions, we develop models that utilize features based on contextualized word representations from PLMs, achieving performance that rivals or surpasses human accuracy. Secondly, to enhance model generalization to unseen labels, we create small finetuning datasets with domain-independent task label descriptions, improving model performance and robustness. Lastly, we tackle the sensitivity of large language models to in-context learning prompts by selecting effective demonstrations, focusing on misclassified examples and resolving model ambiguity regarding test example labels.