Assessing the Performance of Human-Capable LLMs -- Are LLMs Coming for Your Job?

TL;DR

This paper introduces SelfScore, a benchmark for evaluating LLMs on help desk tasks, showing automated agents outperform humans and raising concerns about AI-driven job displacement.

Contribution

The paper develops SelfScore, a new benchmark for assessing LLMs in professional help desk tasks, and demonstrates the effectiveness of RAG-enhanced models over human performance.

Findings

Automated LLM agents outperform human workers on help desk tasks.

Retrieval-Augmented Generation improves domain-specific LLM performance.

SelfScore enables transparent comparison of AI and human agents.

Abstract

The current paper presents the development and validation of SelfScore, a novel benchmark designed to assess the performance of automated Large Language Model (LLM) agents on help desk and professional consultation tasks. Given the increasing integration of AI in industries, particularly within customer service, SelfScore fills a crucial gap by enabling the comparison of automated agents and human workers. The benchmark evaluates agents on problem complexity and response helpfulness, ensuring transparency and simplicity in its scoring system. The study also develops automated LLM agents to assess SelfScore and explores the benefits of Retrieval-Augmented Generation (RAG) for domain-specific tasks, demonstrating that automated LLM agents incorporating RAG outperform those without. All automated LLM agents were observed to perform better than the human control group. Given these results, the study raises concerns about the potential displacement of human workers, especially in areas where AI technologies excel. Ultimately, SelfScore provides a foundational tool for understanding the impact of AI in help desk environments while advocating for ethical considerations in the ongoing transition towards automation.