Automatic Prompt Optimization for Knowledge Graph Construction: Insights from an Empirical Study

TL;DR

This paper empirically evaluates automatic prompt optimization techniques for knowledge graph triple extraction, demonstrating their effectiveness in generating prompts comparable to human-crafted ones and improving extraction performance across various settings.

Contribution

It provides a comprehensive empirical analysis of automatic prompt optimization methods for KG triple extraction, highlighting their potential to reduce manual effort and enhance accuracy.

Findings

Optimized prompts perform comparably to human-crafted prompts.

Prompt optimization improves extraction accuracy with increased schema complexity.

Larger input text size enhances the effectiveness of prompt optimization.

Abstract

A KG represents a network of entities and illustrates relationships between them. KGs are used for various applications, including semantic search and discovery, reasoning, decision-making, natural language processing, machine learning, and recommendation systems. Triple (subject-relation-object) extraction from text is the fundamental building block of KG construction and has been widely studied, for example, in early benchmarks such as ACE 2002 to more recent ones, such as WebNLG 2020, REBEL and SynthIE. While the use of LLMs is explored for KG construction, handcrafting reasonable task-specific prompts for LLMs is a labour-intensive exercise and can be brittle due to subtle changes in the LLM models employed. Recent work in NLP tasks (e.g. autonomy generation) uses automatic prompt optimization/engineering to address this challenge by generating optimal or near-optimal task-specific prompts given input-output examples. This empirical study explores the application of automatic prompt optimization for the triple extraction task using experimental benchmarking. We evaluate different settings by changing (a) the prompting strategy, (b) the LLM being used for prompt optimization and task execution, (c) the number of canonical relations in the schema (schema complexity), (d) the length and diversity of input text, (e) the metric used to drive the prompt optimization, and (f) the dataset being used for training and testing. We evaluate three different automatic prompt optimizers, namely, DSPy, APE, and TextGrad and use two different triple extraction datasets, SynthIE and REBEL. Through rigorous empirical evaluation, our main contribution highlights that automatic prompt optimization techniques can generate reasonable prompts similar to humans for triple extraction. In turn, these optimized prompts achieve improved results, particularly with increasing schema complexity and text size.