From Text to Insight: Leveraging Large Language Models for Performance Evaluation in Management

TL;DR

This paper investigates GPT-4's ability to objectively evaluate knowledge-based performance in management, finding it reliable and consistent but also susceptible to biases, thus highlighting both potential and limitations of LLMs in organizational assessments.

Contribution

It demonstrates that LLMs like GPT-4 can reliably evaluate performance outputs, offering a novel AI-based approach to management performance evaluation.

Findings

GPT-4 ratings are comparable to human ratings

GPT ratings show higher consistency and reliability

LLMs are prone to contextual biases like the halo effect

Abstract

This study explores the potential of Large Language Models (LLMs), specifically GPT-4, to enhance objectivity in organizational task performance evaluations. Through comparative analyses across two studies, including various task performance outputs, we demonstrate that LLMs can serve as a reliable and even superior alternative to human raters in evaluating knowledge-based performance outputs, which are a key contribution of knowledge workers. Our results suggest that GPT ratings are comparable to human ratings but exhibit higher consistency and reliability. Additionally, combined multiple GPT ratings on the same performance output show strong correlations with aggregated human performance ratings, akin to the consensus principle observed in performance evaluation literature. However, we also find that LLMs are prone to contextual biases, such as the halo effect, mirroring human evaluative biases. Our research suggests that while LLMs are capable of extracting meaningful constructs from text-based data, their scope is currently limited to specific forms of performance evaluation. By highlighting both the potential and limitations of LLMs, our study contributes to the discourse on AI role in management studies and sets a foundation for future research to refine AI theoretical and practical applications in management.