Learning to Predict Usage Options of Product Reviews with LLM-Generated Labels

TL;DR

This paper introduces a novel approach using LLMs for annotating product review data to predict usage options, demonstrating cost-effective, high-quality labels that surpass traditional methods and match expert standards.

Contribution

The paper presents a new LLM-based annotation method with a novel evaluation metric, improving data quality and cost-efficiency for complex natural language tasks.

Findings

LLM-generated labels outperform third-party vendor annotations.

GPT-4 labels reach expert-level quality.

The proposed method reduces annotation costs significantly.

Abstract

Annotating large datasets can be challenging. However, crowd-sourcing is often expensive and can lack quality, especially for non-trivial tasks. We propose a method of using LLMs as few-shot learners for annotating data in a complex natural language task where we learn a standalone model to predict usage options for products from customer reviews. We also propose a new evaluation metric for this scenario, HAMS4, that can be used to compare a set of strings with multiple reference sets. Learning a custom model offers individual control over energy efficiency and privacy measures compared to using the LLM directly for the sequence-to-sequence task. We compare this data annotation approach with other traditional methods and demonstrate how LLMs can enable considerable cost savings. We find that the quality of the resulting data exceeds the level attained by third-party vendor services and that GPT-4-generated labels even reach the level of domain experts. We make the code and generated labels publicly available.