Granting GPT-4 License and Opportunity: Enhancing Accuracy and Confidence Estimation for Few-Shot Event Detection

TL;DR

This paper introduces a prompt-based method for GPT-4 that enhances confidence estimation and accuracy in few-shot event detection, enabling more reliable ontology refinement without extra computational overhead.

Contribution

It proposes expanding GPT-4 prompts with 'License to speculate' and 'Opportunity' to improve confidence estimation and accuracy in few-shot event detection tasks.

Findings

Achieved 0.759 AUC in confidence estimation

Improved event detection accuracy without extra machinery

Enabled reliable confidence measures for ontology refinement

Abstract

Large Language Models (LLMs) such as GPT-4 have shown enough promise in the few-shot learning context to suggest use in the generation of "silver" data and refinement of new ontologies through iterative application and review. Such workflows become more effective with reliable confidence estimation. Unfortunately, confidence estimation is a documented weakness of models such as GPT-4, and established methods to compensate require significant additional complexity and computation. The present effort explores methods for effective confidence estimation with GPT-4 with few-shot learning for event detection in the BETTER ontology as a vehicle. The key innovation is expanding the prompt and task presented to GPT-4 to provide License to speculate when unsure and Opportunity to quantify and explain its uncertainty (L&O). This approach improves accuracy and provides usable confidence measures (0.759 AUC) with no additional machinery.