Improving Social Meaning Detection with Pragmatic Masking and Surrogate Fine-Tuning

TL;DR

This paper introduces pragmatic masking and surrogate fine-tuning strategies to improve social meaning detection in social media, achieving significant gains across multiple datasets and languages, especially in few-shot scenarios.

Contribution

It presents novel masking and fine-tuning methods that leverage social cues, outperforming existing models on diverse social meaning detection tasks.

Findings

Achieved 2.34% higher F1 than baseline on Twitter datasets.

Significantly improved few-shot learning performance with only 5% training data.

Demonstrated language-agnostic effectiveness in zero-shot multilingual settings.

Abstract

Masked language models (MLMs) are pre-trained with a denoising objective that is in a mismatch with the objective of downstream fine-tuning. We propose pragmatic masking and surrogate fine-tuning as two complementing strategies that exploit social cues to drive pre-trained representations toward a broad set of concepts useful for a wide class of social meaning tasks. We test our models on $15$ different Twitter datasets for social meaning detection. Our methods achieve $2.34\%$ $F_1$ over a competitive baseline, while outperforming domain-specific language models pre-trained on large datasets. Our methods also excel in few-shot learning: with only $5\%$ of training data (severely few-shot), our methods enable an impressive $68.54\%$ average $F_1$. The methods are also language agnostic, as we show in a zero-shot setting involving six datasets from three different languages.