Concept-Guided Chain-of-Thought Prompting for Pairwise Comparison Scoring of Texts with Large Language Models

TL;DR

This paper introduces CGCoT, a novel method using large language models and concept-guided prompts to score texts by transforming pairwise comparisons into pattern recognition, outperforming traditional unsupervised methods.

Contribution

The paper presents a new framework, CGCoT, that leverages LLMs and human-designed prompts to effectively score texts without extensive labeled data, especially for short texts.

Findings

Stronger correlation with human judgments than Wordfish.

Achieves comparable results to fine-tuned RoBERTa-Large with minimal supervision.

Effective in analyzing politically sensitive Twitter data.

Abstract

Existing text scoring methods require a large corpus, struggle with short texts, or require hand-labeled data. We develop a text scoring framework that leverages generative large language models (LLMs) to (1) set texts against the backdrop of information from the near-totality of the web and digitized media, and (2) effectively transform pairwise text comparisons from a reasoning problem to a pattern recognition task. Our approach, concept-guided chain-of-thought (CGCoT), utilizes a chain of researcher-designed prompts with an LLM to generate a concept-specific breakdown for each text, akin to guidance provided to human coders. We then pairwise compare breakdowns using an LLM and aggregate answers into a score using a probability model. We apply this approach to better understand speech reflecting aversion to specific political parties on Twitter, a topic that has commanded increasing interest because of its potential contributions to democratic backsliding. We achieve stronger correlations with human judgments than widely used unsupervised text scoring methods like Wordfish. In a supervised setting, besides a small pilot dataset to develop CGCoT prompts, our measures require no additional hand-labeled data and produce predictions on par with RoBERTa-Large fine-tuned on thousands of hand-labeled tweets. This project showcases the potential of combining human expertise and LLMs for scoring tasks.