Large Language Models in the Workplace: A Case Study on Prompt Engineering for Job Type Classification

TL;DR

This study evaluates the effectiveness of prompt engineering with GPT-3.5 models for classifying job postings as suitable for entry-level positions, demonstrating superior performance over traditional supervised methods.

Contribution

It introduces a detailed analysis of prompt engineering techniques for LLMs in job classification, highlighting the importance of prompt wording and achieving state-of-the-art zero-shot results.

Findings

Zero-shot GPT-3.5 turbo outperforms supervised models by 6% in Precision@95% Recall.

Prompt wording significantly impacts model performance.

Prompt engineering is crucial for eliciting reasoning in LLMs.

Abstract

This case study investigates the task of job classification in a real-world setting, where the goal is to determine whether an English-language job posting is appropriate for a graduate or entry-level position. We explore multiple approaches to text classification, including supervised approaches such as traditional models like Support Vector Machines (SVMs) and state-of-the-art deep learning methods such as DeBERTa. We compare them with Large Language Models (LLMs) used in both few-shot and zero-shot classification settings. To accomplish this task, we employ prompt engineering, a technique that involves designing prompts to guide the LLMs towards the desired output. Specifically, we evaluate the performance of two commercially available state-of-the-art GPT-3.5-based language models, text-davinci-003 and gpt-3.5-turbo. We also conduct a detailed analysis of the impact of different aspects of prompt engineering on the model's performance. Our results show that, with a well-designed prompt, a zero-shot gpt-3.5-turbo classifier outperforms all other models, achieving a 6% increase in Precision@95% Recall compared to the best supervised approach. Furthermore, we observe that the wording of the prompt is a critical factor in eliciting the appropriate "reasoning" in the model, and that seemingly minor aspects of the prompt significantly affect the model's performance.