This Candidate is [MASK]. Prompt-based Sentiment Extraction and Reference Letters

TL;DR

The paper introduces a prompt-based sentiment extraction method using pre-trained large language models that requires no fine-tuning, effectively analyzing reference letters and revealing gendered language biases impacting job market outcomes.

Contribution

It presents a novel, fine-tuning-free sentiment analysis approach with probability interpretation, applied to reference letters to link sentiment to job market success and gender biases.

Findings

Higher sentiment scores correlate with better job market outcomes.

Sentiment dispersion negatively impacts candidate performance.

Gendered language differences in reference letters influence employment prospects.

Abstract

I propose a relatively simple way to deploy pre-trained large language models (LLMs) in order to extract sentiment and other useful features from text data. The method, which I refer to as prompt-based sentiment extraction, offers multiple advantages over other methods used in economics and finance. In particular, it accepts the text input as is (without pre-processing) and produces a sentiment score that has a probability interpretation. Unlike other LLM-based approaches, it does not require any fine-tuning or labeled data. I apply my prompt-based strategy to a hand-collected corpus of confidential reference letters (RLs). I show that the sentiment contents of RLs are clearly reflected in job market outcomes. Candidates with higher average sentiment in their RLs perform markedly better regardless of the measure of success chosen. Moreover, I show that sentiment dispersion among letter writers negatively affects the job market candidate's performance. I compare my sentiment extraction approach to other commonly used methods for sentiment analysis: `bag-of-words' approaches, fine-tuned language models, and querying advanced chatbots. No other method can fully reproduce the results obtained by prompt-based sentiment extraction. Finally, I slightly modify the method to obtain `gendered' sentiment scores (as in Eberhardt et al., 2023). I show that RLs written for female candidates emphasize `grindstone' personality traits, whereas male candidates' letters emphasize `standout' traits. These gender differences negatively affect women's job market outcomes.