LIFELONG SOTOPIA: Evaluating Social Intelligence of Language Agents Over Lifelong Social Interactions

TL;DR

This paper introduces LIFELONG-SOTOPIA, a benchmark for evaluating the social intelligence of language agents over extended interactions, revealing current limitations in goal achievement and believability compared to humans.

Contribution

The paper presents a novel lifelong social interaction benchmark for language agents and provides comprehensive evaluation results highlighting current performance gaps.

Findings

Goal achievement declines over interactions.

Memory methods improve performance but are insufficient.

Agents lag behind humans in complex social understanding.

Abstract

Humans engage in lifelong social interactions through interacting with different people under different scenarios for different social goals. This requires social intelligence to gather information through a long time span and use it to navigate various social contexts effectively. Whether AI systems are also capable of this is understudied in the existing research. In this paper, we present a novel benchmark, LIFELONG-SOTOPIA, to perform a comprehensive evaluation of language agents by simulating multi-episode interactions. In each episode, the language agents role-play characters to achieve their respective social goals in randomly sampled social tasks. With LIFELONG-SOTOPIA, we find that goal achievement and believability of all of the language models that we test decline through the whole interaction. Although using an advanced memory method improves the agents' performance, the best agents still achieve a significantly lower goal completion rate than humans on scenarios requiring an explicit understanding of interaction history. These findings show that we can use LIFELONG-SOTOPIA to evaluate the social intelligence of language agents over lifelong social interactions.