Evaluating Contrastive Feedback for Effective User Simulations

TL;DR

This paper investigates how contrastive prompt engineering techniques can improve user simulation accuracy using Large Language Models in interactive information retrieval, focusing on the influence of different contextual information modalities.

Contribution

It introduces contrastive prompting methods for LLM-based user simulations and analyzes their impact on simulation effectiveness in information retrieval tasks.

Findings

Contrastive prompts improve simulation accuracy.

Different contextual modalities influence user simulation performance.

Enhanced models better mimic realistic user behavior.

Abstract

The use of Large Language Models (LLMs) for simulating user behavior in the domain of Interactive Information Retrieval has recently gained significant popularity. However, their application and capabilities remain highly debated and understudied. This study explores whether the underlying principles of contrastive training techniques, which have been effective for fine-tuning LLMs, can also be applied beneficially in the area of prompt engineering for user simulations. Previous research has shown that LLMs possess comprehensive world knowledge, which can be leveraged to provide accurate estimates of relevant documents. This study attempts to simulate a knowledge state by enhancing the model with additional implicit contextual information gained during the simulation. This approach enables the model to refine the scope of desired documents further. The primary objective of this study is to analyze how different modalities of contextual information influence the effectiveness of user simulations. Various user configurations were tested, where models are provided with summaries of already judged relevant, irrelevant, or both types of documents in a contrastive manner. The focus of this study is the assessment of the impact of the prompting techniques on the simulated user agent performance. We hereby lay the foundations for leveraging LLMs as part of more realistic simulated users.