USimAgent: Large Language Models for Simulating Search Users

TL;DR

USimAgent leverages large language models to simulate complex user search behaviors, improving query generation and providing a new tool for evaluating information retrieval systems effectively.

Contribution

The paper introduces USimAgent, a novel LLM-based user simulator capable of generating complete search sessions, advancing the realism and utility of user behavior simulation.

Findings

Outperforms existing methods in query generation

Comparable to traditional methods in predicting clicks and stopping behaviors

Validates the effectiveness of LLMs in user simulation

Abstract

Due to the advantages in the cost-efficiency and reproducibility, user simulation has become a promising solution to the user-centric evaluation of information retrieval systems. Nonetheless, accurately simulating user search behaviors has long been a challenge, because users' actions in search are highly complex and driven by intricate cognitive processes such as learning, reasoning, and planning. Recently, Large Language Models (LLMs) have demonstrated remarked potential in simulating human-level intelligence and have been used in building autonomous agents for various tasks. However, the potential of using LLMs in simulating search behaviors has not yet been fully explored. In this paper, we introduce a LLM-based user search behavior simulator, USimAgent. The proposed simulator can simulate users' querying, clicking, and stopping behaviors during search, and thus, is capable of generating complete search sessions for specific search tasks. Empirical investigation on a real user behavior dataset shows that the proposed simulator outperforms existing methods in query generation and is comparable to traditional methods in predicting user clicks and stopping behaviors. These results not only validate the effectiveness of using LLMs for user simulation but also shed light on the development of a more robust and generic user simulators. The code and data are accessible at https://github.com/Meow-E/USimAgent.