Zero-Shot Ranking Socio-Political Texts with Transformer Language Models to Reduce Close Reading Time

TL;DR

This paper presents a zero-shot ranking method using Transformer models to classify socio-political texts as entailment problems, significantly reducing close reading time by ranking documents based on their likelihood of being relevant.

Contribution

It introduces a novel zero-shot ranking approach for socio-political text classification using Transformer models and explores query formulation impacts on performance.

Findings

DeBERTa outperforms RoBERTa in mean average precision.

Declarative queries yield better results than dictionary definitions.

Close reading time can be reduced by selecting top-ranked documents based on desired recall.

Abstract

We approach the classification problem as an entailment problem and apply zero-shot ranking to socio-political texts. Documents that are ranked at the top can be considered positively classified documents and this reduces the close reading time for the information extraction process. We use Transformer Language Models to get the entailment probabilities and investigate different types of queries. We find that DeBERTa achieves higher mean average precision scores than RoBERTa and when declarative form of the class label is used as a query, it outperforms dictionary definition of the class label. We show that one can reduce the close reading time by taking some percentage of the ranked documents that the percentage depends on how much recall they want to achieve. However, our findings also show that percentage of the documents that should be read increases as the topic gets broader.