RED-CT: A Systems Design Methodology for Using LLM-labeled Data to Train and Deploy Edge Classifiers for Computational Social Science

TL;DR

This paper presents RED-CT, a systems design methodology that leverages large language models as imperfect data annotators, with intervention measures to improve classification performance for deploying edge classifiers in social science applications.

Contribution

It introduces a novel systems design approach that effectively integrates LLMs into supervised learning workflows, outperforming LLM-generated labels in most tests.

Findings

Method outperforms LLM-generated labels in 7 of 8 tests.

System intervention measures improve classification accuracy.

Applicable to deploying edge classifiers in social science contexts.

Abstract

Large language models (LLMs) have enhanced our ability to rapidly analyze and classify unstructured natural language data. However, concerns regarding cost, network limitations, and security constraints have posed challenges for their integration into work processes. In this study, we adopt a systems design approach to employing LLMs as imperfect data annotators for downstream supervised learning tasks, introducing novel system intervention measures aimed at improving classification performance. Our methodology outperforms LLM-generated labels in seven of eight tests, demonstrating an effective strategy for incorporating LLMs into the design and deployment of specialized, supervised learning models present in many industry use cases.