Transformer Encoder for Social Science

TL;DR

This paper introduces TESS, a compact transformer encoder tailored for social science text analysis, outperforming BERT and RoBERTa especially with limited training data.

Contribution

The paper presents TESS, a new pretrained transformer model specifically designed for social science text tasks, demonstrating superior performance with small datasets.

Findings

TESS outperforms BERT and RoBERTa by 16.7% on average with limited data.

TESS is specifically optimized for social science text processing.

The model shows robustness in low-data scenarios.

Abstract

High-quality text data has become an important data source for social scientists. We have witnessed the success of pretrained deep neural network models, such as BERT and RoBERTa, in recent social science research. In this paper, we propose a compact pretrained deep neural network, Transformer Encoder for Social Science (TESS), explicitly designed to tackle text processing tasks in social science research. Using two validation tests, we demonstrate that TESS outperforms BERT and RoBERTa by 16.7% on average when the number of training samples is limited (<1,000 training instances). The results display the superiority of TESS over BERT and RoBERTa on social science text processing tasks. Lastly, we discuss the limitation of our model and present advice for future researchers.