Identifying Key Terms in Prompts for Relevance Evaluation with GPT Models

TL;DR

This paper investigates how specific terms in prompts influence the effectiveness of GPT models in relevance evaluation tasks, emphasizing the importance of prompt design and term selection for improved performance.

Contribution

It identifies key terms that enhance relevance evaluation with GPT models and analyzes the impact of prompt wording and examples on evaluation accuracy.

Findings

Using 'answer' improves relevance assessment over 'relevant'

Balancing scope of relevance is crucial for accuracy

Few-shot examples refine relevance criteria

Abstract

Relevance evaluation of a query and a passage is essential in Information Retrieval (IR). Recently, numerous studies have been conducted on tasks related to relevance judgment using Large Language Models (LLMs) such as GPT-4, demonstrating significant improvements. However, the efficacy of LLMs is considerably influenced by the design of the prompt. The purpose of this paper is to identify which specific terms in prompts positively or negatively impact relevance evaluation with LLMs. We employed two types of prompts: those used in previous research and generated automatically by LLMs. By comparing the performance of these prompts in both few-shot and zero-shot settings, we analyze the influence of specific terms in the prompts. We have observed two main findings from our study. First, we discovered that prompts using the term answerlead to more effective relevance evaluations than those using relevant. This indicates that a more direct approach, focusing on answering the query, tends to enhance performance. Second, we noted the importance of appropriately balancing the scope of relevance. While the term relevant can extend the scope too broadly, resulting in less precise evaluations, an optimal balance in defining relevance is crucial for accurate assessments. The inclusion of few-shot examples helps in more precisely defining this balance. By providing clearer contexts for the term relevance, few-shot examples contribute to refine relevance criteria. In conclusion, our study highlights the significance of carefully selecting terms in prompts for relevance evaluation with LLMs.