Dictionary-Assisted Supervised Contrastive Learning

TL;DR

This paper introduces DASCL, a novel method that leverages domain-specific dictionaries to improve supervised contrastive learning for text classification, especially in social science applications with limited data.

Contribution

DASCL is a new approach that incorporates dictionary-based keyword simplification into supervised contrastive learning to enhance model performance in few-shot settings.

Findings

Improves classification metrics in few-shot learning scenarios.

Outperforms traditional cross-entropy and other contrastive methods.

Enhances social science text analysis with domain knowledge integration.

Abstract

Text analysis in the social sciences often involves using specialized dictionaries to reason with abstract concepts, such as perceptions about the economy or abuse on social media. These dictionaries allow researchers to impart domain knowledge and note subtle usages of words relating to a concept(s) of interest. We introduce the dictionary-assisted supervised contrastive learning (DASCL) objective, allowing researchers to leverage specialized dictionaries when fine-tuning pretrained language models. The text is first keyword simplified: a common, fixed token replaces any word in the corpus that appears in the dictionary(ies) relevant to the concept of interest. During fine-tuning, a supervised contrastive objective draws closer the embeddings of the original and keyword-simplified texts of the same class while pushing further apart the embeddings of different classes. The keyword-simplified texts of the same class are more textually similar than their original text counterparts, which additionally draws the embeddings of the same class closer together. Combining DASCL and cross-entropy improves classification performance metrics in few-shot learning settings and social science applications compared to using cross-entropy alone and alternative contrastive and data augmentation methods.