Systematic Evaluation of GPT-3 for Zero-Shot Personality Estimation

TL;DR

This study systematically evaluates GPT-3's ability to estimate Big 5 personality traits from social media posts in a zero-shot setting, revealing strengths and limitations in human-level NLP tasks.

Contribution

It provides a comprehensive analysis of GPT-3's zero-shot performance on personality estimation, highlighting areas for improvement and comparing it to existing models.

Findings

GPT-3 performs close to state-of-the-art in broad classification tasks.

Performance drops to baseline levels in fine-grained classification.

Systematic errors suggest directions for enhancing LLMs in human-centric NLP.

Abstract

Very large language models (LLMs) perform extremely well on a spectrum of NLP tasks in a zero-shot setting. However, little is known about their performance on human-level NLP problems which rely on understanding psychological concepts, such as assessing personality traits. In this work, we investigate the zero-shot ability of GPT-3 to estimate the Big 5 personality traits from users' social media posts. Through a set of systematic experiments, we find that zero-shot GPT-3 performance is somewhat close to an existing pre-trained SotA for broad classification upon injecting knowledge about the trait in the prompts. However, when prompted to provide fine-grained classification, its performance drops to close to a simple most frequent class (MFC) baseline. We further analyze where GPT-3 performs better, as well as worse, than a pretrained lexical model, illustrating systematic errors that suggest ways to improve LLMs on human-level NLP tasks.