AgentSociety Challenge: Designing LLM Agents for User Modeling and Recommendation on Web Platforms

TL;DR

The AgentSociety Challenge is a pioneering competition exploring how Large Language Model agents can improve user modeling and recommendations on web platforms, demonstrating significant performance gains and fostering innovation.

Contribution

This paper introduces the first web conference LLM agent challenge, providing a benchmark environment and analyzing top-performing agent designs for user modeling and recommendations.

Findings

Achieved over 20% performance improvement in both tracks during development.

Secured up to 15.9% improvement in the final phase.

Engaged 295 teams with 1,400+ submissions, indicating high community interest.

Abstract

The AgentSociety Challenge is the first competition in the Web Conference that aims to explore the potential of Large Language Model (LLM) agents in modeling user behavior and enhancing recommender systems on web platforms. The Challenge consists of two tracks: the User Modeling Track and the Recommendation Track. Participants are tasked to utilize a combined dataset from Yelp, Amazon, and Goodreads, along with an interactive environment simulator, to develop innovative LLM agents. The Challenge has attracted 295 teams across the globe and received over 1,400 submissions in total over the course of 37 official competition days. The participants have achieved 21.9% and 20.3% performance improvement for Track 1 and Track 2 in the Development Phase, and 9.1% and 15.9% in the Final Phase, representing a significant accomplishment. This paper discusses the detailed designs of the Challenge, analyzes the outcomes, and highlights the most successful LLM agent designs. To support further research and development, we have open-sourced the benchmark environment at https://tsinghua-fib-lab.github.io/AgentSocietyChallenge.