Leveraging LLMs to Evaluate Usefulness of Document

TL;DR

This paper presents a novel user-centric evaluation framework leveraging LLMs to generate multilevel usefulness labels by incorporating user context and behavior, improving relevance assessment and satisfaction prediction.

Contribution

It introduces a new LLM-based evaluation method that integrates user context and behavioral data for more accurate usefulness assessment, surpassing traditional relevance labeling.

Findings

LLMs can accurately evaluate usefulness with proper context guidance

The approach outperforms third-party labeling methods in usefulness assessment

Use of generated labels improves satisfaction prediction models

Abstract

The conventional Cranfield paradigm struggles to effectively capture user satisfaction due to its weak correlation between relevance and satisfaction, alongside the high costs of relevance annotation in building test collections. To tackle these issues, our research explores the potential of leveraging large language models (LLMs) to generate multilevel usefulness labels for evaluation. We introduce a new user-centric evaluation framework that integrates users' search context and behavioral data into LLMs. This framework uses a cascading judgment structure designed for multilevel usefulness assessments, drawing inspiration from ordinal regression techniques. Our study demonstrates that when well-guided with context and behavioral information, LLMs can accurately evaluate usefulness, allowing our approach to surpass third-party labeling methods. Furthermore, we conduct ablation studies to investigate the influence of key components within the framework. We also apply the labels produced by our method to predict user satisfaction, with real-world experiments indicating that these labels substantially improve the performance of satisfaction prediction models.